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DETAILED ACTION 
Continued Examination Under 37 CFR 1.114 

1. A request for continued examination under 37 CFR 1.114, including the fee set 
forth in 37 CFR 1 .17(e), was filed in this application after final rejection. Since this 
application is eligible for continued examination under 37 CFR 1.114, and the fee set 
forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action 
has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on January 
4, 2006 has been entered. 

Response to Amendment 

2. The amendments to the claims have been entered. Claims 1 , 16, 1 9, 32, 51 , 
and 52 are currently amended and new claims 56-61 have been added. 

Response to Arguments 

3. Applicant's arguments with respect to claims 17, 18, 51 , and 52 have been 
considered but are moot in view of the new ground(s) of rejection. 

Claim Rejections - 35 USC §112 

4. The amendment to claim 16 overcomes the rejection under 35 U.S.C. 112 made 
in the previous Office Action. The rejection of claim 16 under 35 U.S.C. 1 12 is 
withdrawn. 
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Claim Rejections - 35 USC § 103 

5. The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set 
forth in section 102 of this title, if the differences between the subject matter sought to be patented and 
the prior art are such that the subject matter as a whole would have been obvious at the time the 
invention was made to a person having ordinary skill in the art to which said subject matter pertains. 
Patentability shall not be negatived by the manner in which the invention was made. 

6. Claims 1-4, 6-15, 17-22, 24-35, 37-46, and 49-61 are rejected under 35 U.S.C. 
103(a) as being unpatentable over Bejar et al., in view of Walker et al. (U.S. Patent 
5,862,223). 

In regard to claims 1,19, and 32, Bejar et al. disclose a method, storage 
medium, and apparatus comprising a storage medium for a computer system to user 
human assistance in performing tasks, the method comprising: 

automatically and under control of a first computer system (Fig. 1,10, column 4, 
lines 50-53), causing a task (assessment) to be performed by, 

identifying a first and second subtask of the task (the task of evaluating scores 
awarded by human evaluators to constructed responses is broken into subtasks by 
assigning each constructed response to a category, each category representing a 
subtask to be completed, Fig. 1 1A, step 303, column 17, lines 63-67); 

identifying one or more required capabilities of a human for performance of the 
first subtask (each category is an area of expertise to which the human evaluator is 
assigned, step 305, column 17, line 67 to column 18, line 2 and column 8, lines 7-10 
and lines 15-24); 
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dispatching the first subtask to a remote second computer system of a first 
human for performance by said first human, the first human identified as being one or 
more humans who have the capabilities that satisfy the required capabilities of the first 
subtask (step 31 1 , the constructed responses to be evaluated are transmitted to the 
human evaluators at assessment stations, column 18, lines 14-19 and column 5, lines 
24-32); 

receiving a first result from said first human via the second computer system, the 
first result generated by performance of said subtask by said first human (Fig. 1 1 B, step 
315, the constructed responses are scored by the human evaluator and the scores are 
transmitted back from the assessment stations, column 18, lines 30-32); and 

generating a result for said task based at least in part on said first result (step 
321, the scores awarded by the human evaluator are analyzed, column 18, lines 47-51). 

Bejar et al. do not disclose the dispatching step includes providing an indication 
to the first human of a first level of compensation associated with the performance of the 
first subtask, or providing payment to the first human for the performance of the first 
subtask, the provided payment being based on the first compensation level. 

Walker et al. disclose a method and apparatus for providing compensation for 
humans completing subtasks (the grading of tests according to the human's 
qualifications, column 29, lines 15-27). To provide the compensation, the following 
steps are performed: 

dispatching a subtask to a first human, the dispatching including providing an 
indication to the first human of a first level of compensation associated with the 
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performance of the first subtask (a grading request is sent to the qualified graders, 
column 30, lines 6-8; a request includes a price that will be paid to the grader upon 
completion of the grading, column 21, lines 12-26); and 

providing the payment to the first human for performance of the first subtask, the 
provided payment being based on the first compensation level (after completing 
grading, the agreed upon payment is provided to the graders, column 30, lines 35-37 
and column 21 , lines 38-52). 

It would have been obvious to one of ordinary skill in the art at the time of 
invention to modify Bejar et al. to indicate to a first human a level of compensation for 
completing a subtask, then subsequently provide the first human payment 
corresponding with the compensation upon completion of the subtask, in order to 
provide a universally applicable payment protocol for the experts providing advice that 
allows an expert to maximize their earning potential, as taught by Walker et al. (column 
9, lines 19-27). 

In regard to claims 2, 20, and 33, Bejar et al. disclose the identifying by the first 
computer system of the first and second subtasks of the task includes decomposing the 
task into at least the first and second subtasks (the task of evaluating a plurality of 
constructed responses is broken down by assigning the constructed responses to 
categories, column 17, lines 52-67). 
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In regard to claims 3, 21, and 34, Bejar et al. disclose the method further 
comprises the first computer system dispatching said second subtask to a third 
computer system of a second human for performance by the second human (step 307, 
a second human evaluator is assigned to at least one of the same categories as the first 
human evaluator, column 18, lines 2-5), and the first computer system receiving a 
second result from the second human via the third computer system for the second 
subtask (at step 315 when scores from all the evaluators are transmitted, column 18, 
lines 30-32 and lines 35-37); and 

the first computer system further bases its generation of the result for said task 
on said second result (scores of all the evaluators are analyzed, column 18, lines 47- 
51). 

In regard to claims 4, 22, and 35, Bejar et al. disclose said task further 
comprising a third subtask, and the method further comprises the first computer system 
receiving and performing said third subtask producing a third result; and 

the first computer system further bases its generation of the result for said task 
on said third result (Fig. 2B, step 107, a subtask of the overall task of evaluating scores 
awarded by human evaluators to constructed responses includes a subtask of 
determining the correct application to launch for the evaluator, column 7, lines 18-25, 
this subtask is necessary to launch the correct application for the human evaluator, thus 
the generation of the result for the overall task is 'based on' this result, column 8, lines 
51-57). 
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In regard to claims 6, 24, and 37, Bejar et al. discloses said first human is one of 
college educated, at most high school educated, at most elementary school educated, 
and not formally educated (every human is one of these education levels; that is, these 
four education levels represent every level of education a human could possibly have, 
therefore, any human will be one of college educated, at most high school educated, at 
most elementary school educated, and not formally educated). 

In regard to claims 7, 25, and 38, Bejar et al. discloses said subtask is one of 
text, speech, sound, and images related operations (the constructed responses 
evaluated by the human evaluators comprise text documents, audio responses, and 
drawings, column 9, lines 41-50 and column 11, lines 3-5). 

In regard to claims 8, 26, and 39, Bejar et al. discloses the result is one of text, 
numbers, tuples, and sound (column 11, lines 5-7). 

In regard to claims 9, 27, and 40, Bejar et al. discloses said task is one of text 
classification, image comparison, image processing, speech comparison, speech 
recognition, conversion of speech into text, and comparison of music samples (the 
human evaluator classifies the constructed responses by assigned each constructed 
response a score, column 18, lines 23-57). 
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In regard to claims 10, 28, and 41, Bejar et al. discloses said task is associated 
with multiple attributes related to performance of said task, the attributes including an 
accuracy attribute, a security attribute, a timeout attribute, a maximum time spent 
attribute, a maximum cost per task attribute, and a maximum total cost attribute, and 
wherein the identifying of the one or more humans, the dispatching of the first subtask, 
and the generating of the result for said task are performed in a manner to reflect the 
multiple associated attributes (column 10, lines 3-10 and column 13, lines 22-36). 

In regard to claim 1 1 Bejar et al. discloses said task is associated with one or 
more attributes, and said attribute includes an accuracy attribute (column 13, lines 31- 
36). 

In regard to claims 12, 29, and 42, Bejar et al. discloses dispatching said first 
subtask to N-1 additional humans to perform said subtask and said accuracy comprises 
a selection of one of majority governs, and at least N2 agreed answers, wherein N2 and 
N1 are integers, with N2 greater than N1 (column 15, lines 24-28 and column 16, lines 
31-37). 

In regard to claims 13 and 43, Bejar et al. disclose tracking the accuracy of 
additional humans (the accuracy of all evaluators is tracked, column 13, lines 31-36). 
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In regard to claims 14 and 44, Bejar et al. disclose said generation of the result 
further takes into consideration the accuracy of the additional humans (column 19, lines 
23-26). 

In regard to claims 15, 30, and 45, Bejar et al. discloses said task is associated 
with one or more attributes including a security attribute, and said security attribute 
comprises a selection of one of a strict security level, a lax security level, and no 
security level (constructed responses are executed in write protected mode or view only 
mode, column 9, lines 31-41). 

In regard to claim 16, Bejar et al. disclose said task is associated with one or 
more attributes that include a "maximum time" attribute specifying a maximum amount 
of time to be spent by an assigned human to perform said first subtask (elapsed time to 
award a score, column 10, lines 3-7). 

In regard to claims 17 and 18, Bejar et al. do not disclose said task is associated 
with one or more attributes, and said attributes include a maximum cost per task 
attribute or a maximum total task cost attribute. 

Walker et al. disclose associating a task with one or more attributes, wherein the 
attributes include a maximum cost per task attribute (see column 1 1 , lines 40-44, an 
example of criteria provided with a cost per task attribute, i.e. $50 for each solution of at 
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least 10 words, and a maximum total task cost attribute, i.e. $500 for the complete 
solution). 

It would have been obvious to one of ordinary skill in the art at the time of 
invention to modify Bejar et al. to include a maximum cost per task attribute or a 
maximum total task cost attribute, so that an expert's bid for completing a task (see 
column 29, line 64 to column 30, line 8) would not exceed the end user's maximums. 

In regard to claim 31 , Bejar et al. disclose said task is associated with one or 
more attributes, wherein the attributes include a maximum time to be spent on a task 
(elapsed time to award a score, column 10, lines 3-7). 

Bejar et al. do not disclose said attributes include a maximum cost per task 
attribute or a maximum total task cost attribute. 

Walker et al. disclose associating a task with one or more attributes, wherein the 
attributes include a maximum cost per task attribute (see column 1 1 , lines 40-44, an 
example of criteria provided with a cost per task attribute, i.e. $50 for each solution of at 
least 10 words, and a maximum total task cost attribute, i.e. $500 for the complete 
solution). 

It would have been obvious to one of ordinary skill in the art at the time of 
invention to modify Bejar et al. to include a maximum cost per task attribute or a 
maximum total task cost attribute, so that an expert's bid for completing a task (see 
column 29, line 64 to column 30, line 8) would not exceed the end user's maximums. 
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In regard to claim 46, Bejar et al. disclose the first computer system is a task 
server system that is part of a distributed hybrid computer/human computation 
arrangement, and wherein the first human is one of numerous human remote from the 
task seer system who each use distinct client computing devices to act as nodes of the 
distributed hybrid computer/human computation system (Fig. 1, column 5, lines 24-32). 

In regard to claim 49, Bejar et al. disclose the dispatching by the first computer 
system of the first subtask to the remote second computer system is performed using a 
defined application programming interface ("API") (see appendix A). 

In regard to claim 50, Bejar et al. disclose the dispatching by the first computer 
system of the first subtask to the remote second computer system is performed by 
programmatically sending one or more messages from the first computer system to the 
remote computer system (column 5, lines 28-32). 

In regard to claim 51 , Bejar et al. do not disclose providing an indication to the 
first human of the payment to be provided for performance of the first subtask if the 
human chooses to perform the subtask. 

Walker et al. disclose providing an indication to the first human of the payment to 
be provided for performance of the first subtask if the human chooses to perform the 
subtask (a grading request is sent to the qualified graders, column 30, lines 6-8; a 
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request includes a price that will be paid to the grader upon completion of the grading, 
column 21, lines 12-26). 

It would have been obvious to one of ordinary skill in the art at the time of 
invention to modify Bejar et al. to indicate to a first human a payment for completing a 
subtask if the human chooses to perform the subtask, in order to provide a universally 
applicable payment protocol for the experts providing advice that allows an expert to 
maximize their earning potential, as taught by Walker et al. (column 9, lines 19-27). 

In regard to claim 52, Bejar et al. do not disclose the providing of the payment to 
the first human is performed in response to the receiving from the first human of the first 
result from the performance of the first subtask. 

Walker et al. disclose providing of the payment to the first human is performed in 
response to the receiving from the first human of the first result from the performance of 
the first subtask (after completing grading, the agreed upon payment is provided to the 
graders, column 30, lines 35-37 and column 21, lines 38-52). 

It would have been obvious to one of ordinary skill in the art at the time of 
invention to modify Bejar et al. to indicate to a first human a level of compensation for 
completing a subtask, then subsequently provide the first human payment 
corresponding with the compensation upon completion of the subtask, in order to 
provide a universally applicable payment protocol for the experts providing advice that 
allows an expert to maximize their earning potential, as taught by Walker et al. (column 
9, lines 19-27). 
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In regard to claim 53, Bejar et al. disclose the required capabilities of the human 
for performance of the first subtask include an ability to speak a specified language 
(English, column 8, lines 26-29). 

In regard to claim 54, Bejar et al. disclose the required capabilities of the human 
for performance of the first subtask include an ability to hear (constructed response may 
be audio, column 1 1 , lines 3-5). 

In regard to claim 55, Bejar et al. disclose the required capabilities of the human 
for performance of the first subtask include a specified degree of historical accuracy by 
the human when performing subtasks (column 13, lines 31-36). 

In regard to claim 56, Bejar et al. do not disclose the payment provided to the first 
human is further based on part on the quality of the performed subtask. 

Walker et al. disclose payment is further based on quality of the performed 
subtask (column 39, lines 11-15). 

It would have been obvious to one of ordinary skill in the art at the time of 
invention to modify Bejar et al. to base payment to the first human based in part on the 
quality of the performed task, in order to ensure the quality of work provided by the first 
human. 
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In regard to claim 57, Bejar et al. do not disclose the payment provided to the first 
human is based in part on cumulative contributions of the first human. 

Walker et al. disclose payment is based on cumulative contributions of the first 
human (the number of questions graded, column 29, line 59 to column 30, line 8 and 
column 11, lines 40-44). 

It would have been obvious to one of ordinary skill in the art at the time of 
invention to modify Bejar et al. to provide payment based on cumulative contributions of 
the first human, so that the human would be compensated for each subtask performed. 

In regard to claim 58, Bejar et al. do not disclose the payment provided to the first 
human is based on a prior agreement with the first human. 

Walker et al. disclose payment provided to the first human is based on a prior 
agreement with the first human (the expert accepting a subtask is an agreement that the 
price offered is sufficient, column 21, lines 12-29). 

It would have been obvious to one of ordinary skill in the art at the time of 
invention to modify Bejar et al. to provide payment based on a prior agreement so the 
human performing the subtask would be able to decide whether the offered payment 
was sufficient for performing the subtask. 

In regard to claim 59 Bejar et al. do not disclose the first compensation level is a 
first monetary amount and wherein an amount of the payment provided to the first 
human for the performance of the subtask is the first monetary amount. 
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Walker et al. disclose the first compensation level is a first monetary amount (the 
price, column 21 , lines 12-26) and wherein an amount of the payment provided to the 
first human for the performance of the subtask is the first monetary amount (the expert 
is paid the price after completing the subtask, column 21 , lines 38-52). 

It would have been obvious to one of ordinary skill in the art at the time of 
invention to modify Bejar et al. to provide a monetary amount as the first compensation 
level and to provide that monetary amount to the first human after performance of the 
subtask, so the human performing the subtask would be able to decide whether the 
offered payment was sufficient for performing the subtask, and the correct payment 
would be provided to the first human. 

In regard to claim 60, Bejar et al. disclose second subtasks, but do not disclose 
the second subtask has a distinct second monetary amount of compensation and 
providing payment of the secondary amount of compensation. 

Walker et al. disclose the second subtask has a distinct second monetary 
amount of compensation associated with the performance of the second subtask (the 
price of a subtask is based on the qualifications of the expert, column 21, lines 14-18; 
as established above in regard to claim 1 , each category is an area of expertise to 
which the human evaluator is assigned, and each category represents a subtask to be 
completed; since each subtask is associated with a different category, which would 
require different expert qualifications, and the price is based on those qualifications, the 
price of a second category subtask would be distinct from the first category subtask), 
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and wherein the method further comprises providing payment of the second monetary 
amount of compensation to a second human for performance of the second subtask 
(each expert is paid the price after completing the subtask, column 21 , lines 38-52). 

It would have been obvious to one of ordinary skill in the art at the time of 
invention to modify Bejar et al. to associate the second subtask with a distinct 
secondary monetary amount of compensation and to provide that monetary amount to a 
second human, so that the second human would be paid an appropriate amount 
according to their area of expertise. 

In regard to claim 61 , Bejar et al. do not disclose providing the associated 
compensation to the first human for the performance of the first subtask based in part 
on the receiving of the first result. 

Walker et al. disclose the first human is provided the associated compensation 
after the subtask is received (the expert transmits their answer before receiving 
payment, column 21, lines 38-52). 

It would have been obvious to one of ordinary skill in the art at the time of 
invention to modify Bejar et al. to provide the compensation based on the receiving of 
the result, so that the first human was not paid before any actual services were 
rendered. 
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7. Claims 5, 23, and 36 rejected under 35 U.S.C. 103(a) as being unpatentable over 
Bejar et al., in view of Walker et aL, and further in view of Burstein et al. (U.S. Patent 
6,181,909). 

Bejar et al. and Walker et al. do not disclose the first computer system 
additionally performs said second subtask (evaluates a constructed response) and the 
first computer system further bases its generation of the result for said task on said 
second result. 

Burstein et al. disclose a method and system for performing a subtask (scoring 
an essay) to produce a result (essay score, column 3, lines 45-51). 

It would have been obvious to one of ordinary skill in the art at the time of 
invention to modify Bejar et al. and Walker et al. to perform said second subtask 
(scoring of one of the categories of constructed responses) at the first computer using 
the method of automatic essay scoring disclosed by Burstein et al., in order to provide a 
standard score by which the human evaluators could be assessed. 

8. Claims 47 and 48 are rejected under 35 U.S.C. 103(a) as being unpatentable 
over Bejar et al., in view of Walker et al., and further in view of Christensen et al. (U.S. 
Patent 5,881 ,230). 

In regard to claim 47, Bejar et al. discloses that the services of the task server 
system and the coordinating server (i.e. coordinating the performance of the first 
subtask by identifying the first human as having capabilities that satisfy the identified 
required capabilities and sending to the second computer system of the first human an 
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indication of the first subtask to be performed) are performed on the same physical 
server (Fig. 1, assessment system 10, column 17, lines 63-67; column 17, line 67 to 
column 18, line 2; column 8, lines 7-10 and lines 15-24; column 18, lines 14-19; and 
column 5, lines 24-32). 

Bejar et al. and Walker et al. do not disclose that the coordination services are 
performed on a distinct coordination server remote from the task server system, wherein 
the task server system sends information indicating the first subtask and required 
capabilities to the remote coordination server. 

Christensen et al. disclose a method for implementing application objects on 
remote computers as needs dictate so that a given logical model can be implemented 
independently of the underlying physical model (column 13, line 62 to column 14, line 
27). 

It would have been obvious to one of ordinary skill in the art at the time of 
invention to modify Bejar et al. and Walker et al. so the services of the task server and 
the services of the coordinating server were implemented on separate physical servers 
with the appropriate information sent between two, so that the physical performance 
and administration needs of the system could be addressed without giving up the logical 
model, as taught by Christensen et al. (column 13, lines 62-67). 

In regard to claim 48, Bejar et al. disclose the coordinating services identify a first 
and second subtask of the task (column 17, lines 63-67), wherein the coordinating 
services causes the result to be generated for the task based at least in part on the first 
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result (column 18, lines 30-32), wherein the human is one of numerous humans who 
each use distinct client devices to act as nodes available to the coordinating server 
computing system (column 5, lines 24-28). 

Bejar et al. and Walker et al. do not disclose that the coordination services are 
performed on a distinct coordination server remote from the task server system, wherein 
the task server system sends information indicating the first subtask and required 
capabilities to the remote coordination server. 

Christensen et al. disclose a method for implementing application objects on 
remote computers as needs dictate so that a given logical model can be implemented 
independently of the underlying physical model (column 13, line 62 to column 14, line 
27). 

It would have been obvious to one of ordinary skill in the art at the time of 
invention to modify Bejar et al. and Walker et al. so the services of the task server and 
the services of the coordinating server were implemented on separate physical servers 
with the appropriate information sent between two, so that the physical performance 
and administration needs of the system could be addressed without giving up the logical 
model, as taught by Christensen et al. (column 13, lines 62-67). 

Conclusion 

9. Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Brian L Albertalli whose telephone number is (571) 272- 
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7616. The examiner can normally be reached on Mon - Fri, 8:00 AM - 5:30 PM, every 
second Fri off. 
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supervisor, David Hudspeth can be reached on (571) 272-7843. The fax phone number 
for the organization where this application or proceeding is assigned is 571-273-8300. 
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